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Abstract 

In 1993 Carey and Smith conjectured that the most promising way to boost students' 
understanding of the nature of science is a "theory-building approach to teaching about 
inquiry." The research reported here tested this conjecture by comparing results from 
two Grade 4 classrooms that differed in their emphasis on and technological support for 
creating and improving theories. One class followed a Knowledge Building approach 
and used Knowledge Forum®, which together emphasize theory improvement and 
sustained creative work with ideas. The other class followed an inquiry approach 
mediated through collaborative project-based activities. Apart from this, the two 




classes were demographically similar and both fell within the broad category of 
constructivist, inquiry-based approaches and employed a range of modes and media for 
investigative research and reports. An augmented version of Carey and Smith's Nature 
of Science Interview showed that the Knowledge Building approach resulted in deeper 
understanding of the nature of theoretical progress, the connections between theories 
and facts, and the role of ideas in scientific inquiry. 

Resume 

En 1993, Carey et Smith avangaient I'hypothese suivante : la fa?on la plus prometteuse 
de stimuler la comprehension scientifique des eleves etait d'adopter une approche de la 
construction theorique liee a la recherche. La recherche dont il est question dans cet 
article examine cette hypothese en comparant les resultats de deux classes de 
quatrieme annee qui n'avaient pas acces au meme soutien technologique et qui 
n'etaient pas guidees de la meme maniere en matiere de creation et d'amelioration du 
savoir. Une classe a suivi une approche de coelaboration de connaissances et a utilise le 
Knowledge Forum. Ensemble, ils ont mis I'accent sur la construction de theories et ont 
encourage un travail creatif a partir des idees avancees. L'autre classe a adopte une 
approche basee sur la recherche pour mener des projets collaboratifs. A part ces 
differences, les deux classes etaient demographiquement similaires et adoptaient une 
perspective constructiviste, fondee sur des approches basees sur la recherche et 
I'emploi d'une gamme de modes de fonctionnement et de medias pour la recherche et 
la production de comptes rendus. Une version longue de I'entrevue de Carey et Smith, 
Nature of Science, a montre que I'approche de coelaboration de connaissances entraine 
une comprehension approfondie de la nature du progres theorique et du role des idees 
dans la recherche scientifique, et etablit des liens entre theories et faits. 

Introduction 

Science creates new knowledge; that is why its importance is rising, and that is why scientific 
literacy is no longer considered an attribute of a privileged class, but a societal necessity. 
Creation of scientifically literate citizens-who will be able to understand the scientific 
enterprise, to participate intelligently in political decisions relating to science and to produce 
conclusions based on scientific evidence-remains a desirable goal for most educational systems 
(see for instance Council of Canadian Academies, 2010). In addition to political issues, scientific 
literacy plays an important role in the professional development and career choice of citizens. 
Indeed, in the modern rapidly changing world, there is an increase in the number of jobs that 
require highly advanced skills, such as fast learning, creative thinking, and effective production 
of knowledge on a daily basis. According to the National Research Council of the USA (1996), 
development of these skills is closely related to an understanding of science and scientific 
progress. 

Understanding the nature of science and scientific progress continues to be a challenge for 
students at the elementary and middle school levels (Bybee, 2008; DeBoer, 2000; Hodson, 

2003; Laugksch, 2000). Typically, students do not perceive science as a creative, idea-driven 
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enterprise. They are more likely to see it as the methodical collection of observations and 
evidence (Carey & Smith, 1993; Smith, Maclin, Houghton, & Hennessey, 2000). Such a limited 
conception provides little motivation to pursue scientific careers, to support investment in 
science, or to make wise use of its findings. Indeed, a career in science is often not considered 
an attractive one (Fawcett, 1991; Lipsett, 2008). Thus, the question becomes: What are the 
pedagogical approaches that would be the most effective for raising levels of scientific literacy? 
This research tested an explicit and fully developed theory-building approach: Knowledge 
Building, which along with Knowledge Forum® technology, encourages students to take high- 
level responsibility for inquiry, and focuses on collective formulation of questions, idea 
development, experimental set-ups, and most importantly, theory creation and theory 
improvement (Scardamalia, 2004; Scardamalia & Bereiter, 2003, 2006). 

Science as Construction of Ever-Deeper Explanations of the Natural World 

Research on students' conceptions of the nature of science has taken different directions. The 
most widely known direction has concerned students' understanding of the tentative nature of 
empirically based truth claims (meaning that the claims could be modified if new evidence 
comes along) and uncertainty in investigation (Lederman & O'Malley, 1990; Metz, 2004). 
Another course has concerned understanding the relation between hypothesis and evidence, 
the value of negative evidence, and the importance of control of variables (Kuhn, Schauble, & 
Garcia-Mila, 1992; Schauble, Glaser, Duschl, Shulze, & John, 1995). Although both kinds of 
understanding are of clear value as components of scientific literacy, they fall short in 
important respects: They make science out to be a rather plodding process of confirming or 
disconfirming already existing ideas. They do not capture the creative flavour of scientific 
research. Thus, they do not make science out to be a very attractive career choice. 
Furthermore, instruction aimed at these understandings may create a bias in favour of "not 
proven," which is increasingly the position taken by those resisting action on pressing societal 
problems. 

There is another, less developed direction of research on understanding the nature of science, 
which focuses on the role of theories in the creation of scientific knowledge, recognizing the 
goal of science as the construction of ever-deeper explanations of the natural world (Carey & 
Smith, 1993; Driver, Leach, Millar, & Scott, 1996; Smith et al., 2000). That is the direction taken 
in the research reported here. Arguably, an understanding of how new scientific knowledge is 
created carries with it the two other kinds of understanding—the tentative nature of 
empirically-based knowledge and the relation between hypothesis and evidence—but embeds 
them in a more complete and accurate picture of what science is actually about. 

Three Levels of Understanding 

Carey and Smith (1993; see also Smith et al. 2000, and Smith & Wenk, 2006) identified three 
levels related to understanding the nature of science. At the first level of understanding, 
scientific knowledge consists of a simple collection of facts, such as "how to do something 
correctly" or describing/retelling "what happens." At this level there's no clear differentiation 
between theories, hypotheses, or experimental results nor recognition of the role of ideas in 
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scientific inquiry. More precisely, there's little evidence of understanding that activities and 
experiments are the result of ideas that are generated by scientists and that their ideas guide 
scientific investigation. Instead, scientists tend to be seen as individuals doing tests and 
observing what happens, with the resulting accumulation of facts constituting scientific 
progress. Students at this level tend to believe that the resultant knowledge is certain and that 
there is only one objective reality. 

At the second level, scientific knowledge is no longer considered a collection of facts but of 
tested ideas. Scientists do experiments to see if their idea is right, and abandon or revise this 
idea if it is proven wrong. Thus, the guiding role of ideas in experimentation is recognized, with 
ideas clearly differentiated from experimental evidence. Two important notions that appear at 
this level are "explanation" and "hypothesis testing," and two main questions of preoccupation 
are "how things work" and "why things happen." Despite this emergent awareness of the role 
of explanations in scientific progress, there is still no clear understanding of the role of theory in 
framing research and no distinction between theory and hypothesis. 

At the third level, scientific knowledge is represented in theories about the world-theories that 
should not only explain phenomena but also predict them. As Smith et al. (2000) put it: "A 
theory is understood as a coherent, explanatory framework that consists of a network of 
hypothetical theoretical entities that are used to explain patterns of data" (p. 357). At this level, 
individuals understand that theory guides various phases of scientific inquiry, from the 
formulation of hypotheses to the interpretation of results. Rather than judging theories as 
"wrong" or "right," scientists see them as "more or less useful" frameworks for explanation of 
certain phenomena. Therefore, even if level 3 understanding recognizes the rigorous character 
of the scientific process, it is nevertheless aware of the uncertainty of scientific knowledge. 

According to Carey and Smith (1993), the three levels of epistemological understanding form a 
developmental sequence, where level 1 would be typical for the elementary school students 
and level 3 for the advanced graduate students and expert scientists. To assess these levels, the 
authors used a Nature of Science Interview (Carey, Evans, Honda, Jay, & Unger, 1989), 
consisting of explicit questions about the goals of science, the nature of experiments and 
change processes. By means of this instrument, it was shown that traditional pedagogy could 
not be relied on to develop sophisticated epistemologies in students. In traditional science 
classrooms most of the 7 th grade students did not exceed level 1 understanding (Carey et al., 
1989; Honda, 1994; Smith et al., 2000), and the overall average score of the 11 th graders was 
only 1.39 (Honda, 1994). This means that by the end of high school, the majority of students 
still possess an unproblematic epistemology of science and do not recognize the role of theories 
in scientific progress. Similar results demonstrating students' progressions of understanding 
were obtained by other researchers and through other instruments (e.g., Abd-El-Khalick & 
Lederman, 2000; Driver et al., 1996). 
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Three Problems of Understanding 

So what needs to be understood by students in order to perceive scientific progress as a 
construction of ever-deeper explanations of the natural world, and therefore, to move closer to 
the highest level of scientific literacy (level 3 according to the Carey et al. framework)? Three 
main problems of understanding could be identified in this regard. 

The first problem is related to the differentiation between theories and facts. Kenneth Miller 
(2000), pointed out the pervasive confusion about the meaning of the terms "theory" and 
"fact," explaining: 

Theories are not speculative hunches that may some day become "facts" when 
scientists gather enough evidence for them. Theories don't become facts, theories 
explain facts. This means that in scientific terms, theories actually present a higher level 
of understanding than facts." (Worksheet #3 section, para. 1) 

To illustrate the importance of understanding the distinction between these terms, the 
example of the famous Dover, Pennsylvania trial 1 could be useful. This case involved deep 
issues about the nature of scientific knowledge, questioning whether Intelligent Design should 
be taught along with the teaching of evolution. Advocates of including the teaching of 
Intelligent Design have based their case on the claim that evolution is "just a theory," which 
they take to mean "unproven," and which puts it on a par with alternative "unproven theories." 
Some defenders of evolution have responded by denying that it is "just a theory," claiming 
factual status, thus making their position vulnerable to every item of negative or missing 
evidence the Intelligent Design advocates can produce. From Miller's standpoint cited above- 
which eventually won over the judge in the trial-the issue is not whether species evolution is a 
fact but whether Intelligent Design is a scientific theory. Can elementary school students 
understand Miller's subtle but vitally important point? 

The second problem concerns the nature of theoretical progress, notably "How are scientific 
theories improved?" On one hand, theories are judged stronger if they explain more facts—as, 
for instance, Newton's cosmology explained a wider range of astronomical observations than 
did Galileo's. On the other hand, a weak theory such as Intelligent Design may purport to 
explain everything. Predictive power has served as an additional criterion to separate stronger 
from weaker theories. These and other criteria have been woven together into the "theory of 
explanatory coherence" (Thagard, 1989, 2007), which takes account of both logical coherence 
and coherence with empirical observations. Continual movement between empirical evidence 


1 Dover, Pennsylvania, trial (Case No. 04cv2688, The United States District Court for the Middle District of 
Pennsylvania, July 27, 2005) was brought in a United States federal court against a public school district that 
proposed to teach the "theory of Intelligent Design" as a scientific alternative to Darwin's evolution. According to 
Intelligent Design, an intelligent cause can better explain certain aspects of the universe and of living beings, as 
compared to an undirected and random process like natural selection (Harris & Calvert, 2003). It was successfully 
argued by the plaintiffs that intelligent design is a form of creationism, and that the school board policy therefore 
violated the Establishment Clause of the First Amendment to the United States Constitution. 
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and theoretical hypotheses is needed in order to improve scientific theories and to develop a 
higher level of understanding (Klahr & Dunbar, 1988; Zhang, Chen, Sun, & Reid, 2004). Thus 
there is much more to theoretical progress than the amassing of new information. 

The final problem of understanding refers to the discovery process. Where do theories come 
from? How do people invent things? What happens when scientists get stuck? The answers to 
these questions are closely connected to understanding the role of ideas in scientific inquiry. As 
Bereiter and Scardamalia (2008) state "no matter how they are formulated, explanations are 
structures of ideas" (p. 17). Indeed, the "idea" lies at the core of any scientific theory, model, or 
invention. In absence of ideas, no new theoretical framework could be created, no new model 
could be proposed (which is often the reason for "being stuck" in science). Therefore, an ability 
to recognize the role of ideas in creating new knowledge seems to be an important (if not the 
most important) aspect in understanding the nature of science. How can young students grasp 
this? The Knowledge Building approach, developed below, proposes to tackle this problem by 
directly engaging the students in the same process of idea generation and improvement as 
scientists typically do. 

Testing a Theory-Building Approach to Teaching the Nature of Science 

Carey and Smith (1993) conjectured that the most promising way to boost students' 
understanding of the nature of science is a "theory-building approach to teaching about 
inquiry." They admitted, however, that there was little evidence to support this conjecture. The 
present research was designed to test their conjecture by comparing results from two 
classrooms that were similar in most respects but that differed in the amount of emphasis on 
and support for theory building. The study is a "natural experiment" (Dunning, 2008), in that 
the treatments were not assigned but already existed in the practices of the two schools 
participating in the study. A weakness of such an experiment is that control of variables is 
imperfect, but a compensating strength is that the treatments are more realistic and better 
implemented. The group designated as "experimental" pursued theory building and used 
Knowledge Forum® as a normal part of their school's Knowledge Building approach (Bereiter & 
Scardamalia, present issue). The group designated as "comparison" was not only similar 
demographically but also followed an inquiry approach that placed emphasis on collaborative 
research on student-generated questions. Missing from the comparison classroom, however, 
was a focus on generating and improving theories related to research questions and use of a 
supportive electronic forum in which students' ideas were presented in a community space 
with students working collaboratively to advance them. As elaborated below and in other 
articles (see Laferriere, et al.. Moss & Beatty, present issue) students in Knowledge Building 
classrooms frequently set forth theories and try to improve them, and Knowledge Forum 
includes scaffolds that support theory development. Thus, within the limits of a "natural 
experiment," this research represented a controlled test of the variable "support for theory 
building." 

Benefits of Knowledge Building in a number of aspects of scientific thinking have been 
demonstrated (e.g., Bereiter, Scardamalia, Cassells, & Hewitt, 1997; Resendes & Chuy, 2010; 
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Scardamalia; Bereiter, & Lamon, 1994; Zhang, Scardamalia, Reeve & Messina, 2009; Zhang, 
Scardamalia, Lamon, Messina, & Reeve, 2007), but the present study is the first to 
systematically investigate effects of Knowledge Building pedagogy on understanding the nature 
of science. A number of researchers have urged that educational practices directed towards 
interest in science should be started at early grades (Catsambis, 1995; Farenga & Joyce, 1999; 
Patrick, Mantzicopoulos, & Samarapungavan, 2009; Reid, 2003). The present study reflects a 
belief that elementary school science education should also advance students' understanding 
of the nature of science and scientific progress. 

Hypotheses 

We expected that students pursuing a theory-building approach would demonstrate higher 
scientific literacy levels than those of the comparison class. More particularly, we hypothesized 
that Grade 4 students from the experimental class would exceed level 1 epistemological views 
of science (unproblematic accumulation of facts), and exhibit level 2 conceptions such as 
understanding the role of ideas in knowledge acquisition (note that previous studies showed no 
evidence of level 2 conceptions before Grade 6; cf. Smith et al., 2000). 

Method 

Participants 

Participants in the study were nine and ten year-old students from two Grade 4 classes in 
Canada. Although the school where science education was being conducted according to a 
Knowledge Building approach (experimental class) is co-educational, the closest match to it in 
terms of demographics and general constructivist approach to learning was a girls-only school 
(comparison class). For comparability, only data on girls from the experimental class were used. 
The study sample consisted of girls for whom we had signed parental consent, nine from the 
experimental class and 10 from the comparison class. 

The schools typically introduced students to their respective pedagogies in Kindergarten or the 
early years of the elementary school. In the experimental class, the students have entered the 
school in the following grades: pre-school (three girls). Grade 2 (three girls) and Grade 4 (three 
girls). In the comparison class, the entry grades were as follows: Junior Kindergarten (three 
girls). Grade 3 (six girls) and Grade 4 (one girl). Since the study began in the middle of the school 
year, all students experienced respective pedagogies for at least four months, and most of the 
students experienced them for at least two years. 

Pedagogies 

Theory building is inherent in the Knowledge Building approach and serves to support 
Knowledge Building principles such as "Real Ideas, Authentic Problems," "Improvable Ideas," 
"Epistemic Agency," and "Community Knowledge" (Scardamalia & Bereiter, this issue). As 
indicated above. Knowledge Building technology was integral to the work in the experimental 
classroom. The software. Knowledge Forum, is described in a number of articles in this special 
issue (see Gan, Scardamalia, Hong, & Zhang, and Moss & Beatty, present issue; also see 
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Scardamalia, 2004). Here we simply highlight the scaffold feature, as this most directly supports 
theory development. Scaffolds in Knowledge Forum are customizable; the theory development 
set often used in the experimental class is: My Theory, I need to understand. New information. 
This theory cannot explain, A better theory. Evidence, Putting our knowledge together. To use 
scaffolds students simply click on one of the scaffold supports (i.e., one of the phrases in italics 
above) displayed to the left of their note-writing space. The phrase then appears in their note 
and they can continue to generate text or wrap previously generated text around it. The 
scaffold support then serves as a searchable parameter for notes. The following is an excerpt 
from student notes that illustrates how the scaffolds support theory-related efforts. 

I need to understand what is the difference between a reflection and a shadow? 

My theory is that the difference between a shadow and a reflection is a shadow is 
always black. We know that light travels in straight rays. So, a shadow is when we block 
the light. Light cannot bend around an opaque object or go through one. When light is 
blocked, a shadow is made. Reflection is different. When light is reflected it bounces off 
a shiny object like a mirror and changes its direction. 

As suggested by this discourse, the theory-development scaffold encourages users to state their 
theories and what they need to understand to improve those theories. Through participating in 
online and face-to-face discussions in science classes, students developed theories to explain 
various phenomena, formulate questions, explore authoritative sources, answer the questions 
they raise, design experiments to test their theories, analyze the results of experiments, 
reformulate their theory, and so forth. During class discussions teachers encouraged students 
to elaborate their ideas and means to improve them; they do not define tasks in advance, make 
decisions for the students, or present solutions. Thus, the cognitive responsibility for advancing 
knowledge—both their own and that of the community—remains under student control. 
Accordingly, Knowledge Building, and the Knowledge Forum technology that supports it, 
provides support for Knowledge Building principles such as "Real Ideas, Authentic Problems," 
"Improvable Ideas," "Epistemic Agency," and "Community Knowledge, Collective 
Responsibility." 

The comparison school had a six-year history of pursuing a project-based inquiry model of the 
kind elaborated, for instance, by Marx, Blumenfeld, Krajcik and Soloway (1997). Teachers in 
junior and senior kindergarten and Grade 1 were the first to experiment with inquiry as they 
engaged in an in depth study of the Reggio Emilia Philosophy, which emphasizes the 
importance of listening to children's ideas and interests and designing curriculum that responds 
to and challenges these interests (Edwards, Gandini & Forman, 1998). This resulted in a project- 
based approach to inquiry where projects arose from students' questions and projects were 
frequently long-term and open-ended. Teachers invited students to investigate and express 
their ideas through a range of media and modalities. In the project approach drawing, painting, 
sculpting, music-making, movement, acting and other forms of expression were understood as 
tools of investigation, problem-solving and thought (Katz & Chard, 2000). As a result, the study 
of science concepts was frequently integrated with other curriculum. 
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Assessment: Augmented Nature of Science Interview 

Scientific literacy level was assessed through an augmented version of the Nature of Science 
Interview. The interview was composed of two main parts. Part I consisted of the 21 questions 
originally designed by Carey, Smith and their colleagues (see Smith et al., 2000 for the version 
used in this study). These questions were preserved intact to enable comparison with other 
research. Part II consisted of 43 new questions. The new questions tackled different aspects of 
scientific progress, including theory-fact understanding (e.g., "What's the connection between 
theories and facts?"), the role of ideas in the scientific inquiry (e.g., "How do scientists know 
what they are looking for?"), explanatory coherence (e.g., "Say it was your job to compare two 
theories. What would you do to determine if one is better?"), scientific progress (e.g., "Are 
there things we understand today that we didn't understand years ago?"), invention (e.g., 

"How do people invent things?") and finally, on the conception of absolute truth in science 
(e.g., "Some say that scientists can never discover the absolute truth, but they can keep getting 
closer and closer. What do you think about that?"). The new interview questions were pilot- 
tested with students similar to those who participated in this study and revised as needed to 
ensure that the questions were comprehensible to young students and would produce 
responses that could be scored. 

Procedure 

All participants received the Augmented Nature of Science Interview in the beginning of the 
study. Each student was interviewed individually outside of the classroom by a researcher for 
about 40 minutes. Then, after a period of approximately four months, during which the two 
classes studied similar content areas in science (i.e., light energy 2 ) under different teaching 
practices, the augmented Nature of Science Interview was re-administered. Each pre- and post¬ 
interview was video-recorded, and then fully transcribed. 

Scoring of the Augmented Nature of Science Interview 

General scientific literacy level of students was assessed using a multiple trait coding 
procedure, where the interview as a whole was considered in assigning scores on four different 
traits. The multiple trait coding procedure made it possible to follow a student's reasoning from 
the beginning to the end of the interview and base inferences about different aspects of 
scientific literacy on responses to any question providing relevant information. For instance, a 
student's response to the question "Do scientists ever change their ideas?" could provide 
information for understanding the nature of change processes and also awareness of the role of 
ideas in scientific inquiry, goals of science, and relationships between ideas and facts. 

Four traits were defined for analysis, based on the problems of understanding discussed earlier 
in this article: 1—Nature of theoretical progress, 2—Theory-fact understanding, 3—Role of 
ideas in scientific inquiry, and 4—Invention. The first three traits regroup scientific literacy 
aspects that have been identified already by Carey, Smith and their colleagues (Carey et al.. 


2 See Grade 4 Ontario Curriculum, 2007 for "Understanding Matter and Energy": Light and Sound" unit. 


Understanding the nature of science and scientific progress 


9 



1989; Carey & Smith, 1993; Smith et al., 2000). The "nature of theoretical progress" trait 
includes the definition of the goals of science, the role of experiments and theories, how 
scientists make progress and how they decide between competing theories. The "theory-fact 
understanding" trait involves the definitions of the terms such as "theory," "hypothesis," "fact," 
and "experimental result," but also focuses on the connectedness between these terms by 
showing how students deal with the situation when facts disagree with their ideas. The third 
trait is defined around the role of ideas in scientific inquiry with special focus on the role of the 
theoretical framework in the discovery process, and the way it explains why scientists get stuck 
during their work. In addition to the three traits mentioned above, we added a fourth trait, 
called "invention." This trait focuses on the process of invention-creation and the role that 
ideas and theories play in this process. We included this fourth trait because invention has a 
particular significance for societal progress, and it is crucial for scientifically literate citizens to 
understand how innovative things are created. 

Each trait was coded according to the three general levels of understanding of science, as 
defined by Carey, Smith and their colleagues. Thus, the nature of theoretical progress (Trait 1) 
could range from level 1, in which the goals of science are described simply in terms of concrete 
activities (i.e., doing things, gathering information), to level 3, in which the goals of science are 
defined as construction of ever-deeper explanations of the natural world. In the same manner, 
theory-fact understanding (Trait 2) could vary from level 1, with no clear understanding of how 
theory is related to facts, to level 3, with a deep awareness that theory should be evaluated in 
light of multiple results. The role of ideas (Trait 3) could range from level 1, where discoveries 
are described as lucky accidents with no mention of ideas, to level 3, where discovery is a 
complex process guided by theory. Finally, invention (Trait 4) could vary from level 1, in which it 
is defined as a one-step "someone made something" process to level 3, in which invention is 
described as a highly creative multi-step process comprised of producing new forms, 
compositions of matter, or devices. 

This study was part of a larger study including 78 interviews in total. Three independent raters 
scored all the interviews according to the four traits mentioned above using a double-blind 
procedure. If the raters hesitated between two levels, they could allocate an intermediate level 
(e.g., 1.5 or 2.5). A quarter of the interviews were scored by all three coders. An inter-rater 
reliability was calculated through the Cohen's Kappa coefficient that showed substantial level of 
agreement between three raters: k = .64, p < .001 for raters 1 and 2, k = .75, p < .001 for raters 
2 and 3, and finally k = .75, p < .001 for raters 1 and 3. The remaining three-quarters of the 
interviews were equally distributed between three raters for individual coding. For the mutually 
coded set of interviews (a quarter of the full set), the average of the scores obtained by the 
three independent raters was used for further analysis. 

At the end, each student was given four scores, corresponding to each scientific literacy trait: 
1—Nature of theoretical progress," 2—Theory-fact understanding, 3—Role of ideas in scientific 
inquiry and 4—Invention. Finally, four scores were averaged to obtain a mean score indicating 
the general scientific literacy level. 
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Results and Discussion 


A 2 (comparison class vs. Knowledge Building class) x 2 (pre-interview vs. post-interview) 
ANOVA was conducted on each measure presented below, with Class as a between-subjects 
variable and Interview as a within-subjects variable. Table 1 contains means and standard 
deviations for each measure. 


Table 1: Means Scores and Standard Deviations Obtained in Experimental and Comparison 
Classes for General Scientific Literacy Level, Nature of Theoretical Progress (Trait 1), Theory- 
Fact, Understanding (Trait 2), Role of Ideas (Trait 3) and Invention (Trait 4). 


Group 

General 

Level 

Trait 1 

Trait 2 

Trait 3 

Trait 4 

M 

SD 

M 

SD 

M 

SD 

M 

SD 

M 

SD 

Experimental Class 











Pre-Interview 

1.51 

.25 

1.33 

.38 

1.25 

.43 

1.67 

.25 

1.78 

.36 

Post-Interview 

1.66 

.26 

1.69 

.24 

1.42 

.50 

1.81 

.39 

1.72 

.36 

Comparison Class 











Pre-Interview 

1.11 

.30 

1.00 

.33 

.75 

.47 

1.12 

.38 

1.58 

.44 

Post-Interview 

1.33 

.37 

1.30 

.35 

1.08 

.59 

1.30 

.42 

1.65 

.53 


General Scientific Literacy Level 

General scientific literacy level was measured as a mean of the scores obtained for four traits. 
According to our expectations, the general level was significantly higher in the Knowledge 
Building class than in the comparison class, F(l, 17) = 7.59, p < .01 (M = 1.58, SD = .26. vs. M = 
1.22, SD = .35). Thus, Knowledge Building pedagogy led to the development of more 
sophisticated epistemologies in students. Also, both classes showed a general increase in 
scientific literacy from the pre-interview to the post-interview, F(l, 17) = 17.93, p < .001 (M = 
1.30, SD = .34 vs. M = 1.49, SD = .36). However, no Class x Interview interaction was observed, 

F( 1, 17) < 1. 

Closer examination of the scores revealed that already in the beginning of the study Knowledge 
Building students demonstrated higher scientific literacy levels than comparison group 
students, with all students exceeding level 1, and four students being above level 1.5. As for 
students from the comparison class, three of them scored below level 1 and none of them 
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exceeded level 1.5. Wilcoxon-Mann-Whitney test showed a marginally significant difference 
between experimental and comparison groups on the pre-test (U = 74, p = .06). 

In four months, both classes gained about 0.2 points in scientific literacy level. This led to the 
point where the majority of Knowledge Building students moved close to level 2 understanding 
(with scores ranging from 1.6 to 2.0), with 1 student fully reaching this level. As for the 
comparison class, there were two students still below level 1. Only two students exceeded level 
1.5, and none of them reached level 2. Wilcoxon-Mann-Whitney test showed a significant 
difference between experimental and comparison groups on the post-test (U = 77.5, p < .05). 

As shown in Table 1, post-test scores of the comparison group were still lower than the pre-test 
scores of the experimental group. Thus, prior experience in Knowledge Building pedagogy and 
Knowledge Forum technology might be credited with bringing the experimental class to a 
higher-level of understanding of the nature of science, with evidence of level 2 epistemologies. 
To clarify where this pedagogy was especially beneficial to students, each scientific literacy trait 
was examined separately. 

Trait 1: Nature of Theoretical Progress 

The scientific literacy level for the nature of theoretical process, showed a significant Class 
effect, F(l, 17) = 7.54, p < .01 favouring the Knowledge Building class (M = 1.51, SD = .36 over 
the comparison class (M = 1.15, SD = .37). A main effect for the Interview factor was also 
observed, F(l, 17) = 20.24, p < .001, indicating a significant increase from the pre-interview (M 
= 1.16, SD = .39) to the post-interview (M = 1.49, SD = .36) for both classes. Class and Interview 
factors did not interact, F(l, 17) < 1, indicating no significant difference in gains between the 
two classes. 

So, what are the aspects where Knowledge Building and comparison classes especially differ? 

To address this matter, students' answers pertaining to Trait 1 were examined at a more 
detailed level. Although several differences were noted during the pre-test, the most striking 
one was related to understanding the role of experiments in theoretical progress. Comparison 
class tended to explain experiments in terms of procedures, whereas experimental class 
students mostly defined them as providing new knowledge and new understanding. Consider, 
for example, the following answers given by average-scoring comparison students to the 
questions "What is an experiment?" and "Why do scientists do experiments?": 

Experiment like maybe if I add baking soda it will explode or something. 

Experiment is like trying something if it's gonna work 
Below are the examples of answers provided by average-scoring Knowledge Building students: 

An experiment is when you test your ideas. It allows you to get deeper into your ideas 
and research. 

Developing an experiment is a way to help to build your thinking. 
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A notable change in epistemologies from pre- to post-test in the Knowledge Building class is 
associated with how theories are improved. Pre-test responses reflected a belief that theory 
advancement is a matter of accumulating information to get closer to the goal or truth (e.g., 
"Scientists find out more about it [theory] and then they get closer to their goal"). At the time 
of the post-test students conveyed the explanatory role of theories, improvements in methods, 
and important role of experimentation associated with a progressive approach: "Theories are 
getting better by having better technology, experimenting and explaining better the facts." 

Trait 2: Theory-Fact Understanding 

Analysis of Trait 2 showed a marginal Class effect, F( 1, 17) = 3.68, p = .07, with the Knowledge 
Building class (M = 1.33, SD = .46) scoring higher than the comparison class (M = .91, SD = .55). 
There was also a main effect of Interview factor, F(l, 17) = 11.08, p < .01, indicating that theory- 
fact understanding significantly increased from the pre-interview (M = .99, SD = .51) to the 
post-interview (M = 1.24, SD = .56). However, Class x Interview interaction was not significant, 
F(l, 17) < 1, indicating that there was no significant difference in gains between the who 
classes. 

Closer examination of students' answers pertaining to Trait 2 showed that, in contrast to the 
comparison class. Knowledge Building students showed clear theory-fact differentiation from 
the very beginning of the study. When asked to define the terms "theory," "fact" and "idea," 
the type of answers given by students from the experimental and comparison classes differed 
considerably. The following are typical comparison class answers: 

Theory it's almost like a fact. 

Fact it's not like a common thing that everybody knows. 

Ideas you don't know them , facts you already know them. 

Typical Knowledge Building class answers were: 

Theories come from questions. Facts come from experiments. 

Idea is the basis, you're always working on it. Facts help you understanding the idea or 
build on information. 

Ideas allow to make predictions about what might happen in the experiment. 

Thus, in the Knowledge Building class, students were not only able to differentiate and define 
the terms, but also to make relevant connections between them. The differentiation between 
terms became sharper and connections between them more explicit during the post-test: a 
characteristic comment from Grade 4 girls in the Knowledge Building class was that if results 
are not as expected the theory should be re-evaluated. Some additionally noted that the theory 
should be evaluated in light of results from multiple experiments (e.g., "If facts did not agree 
with my theory, I want to do an experiment or two experiments"). 

Pedagogical supports built into Knowledge Forum seem to play and important role in promoting 
a theory improvement process by helping students engage in discourse about theories. 
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evidence, and better theories, and through this process come to differentiate and use scientific 
terms more appropriately. Further efforts, as signalled by the theory-building support putting 
our knowledge together seem to support "explanatory coherence" (cf. Thagard, 1989, 2007). 
Through their discourse students address questions such as "How well is a set of facts explained 
by a particular theory?" "How coherent is the explanation?" "Are there any facts that still need 
to be explained?" This discourse is additionally facilitated by teachers who encourage students 
to address these matters during face-to-face discussions, and then record information collected 
from authoritative sources and classroom experimental work in Knowledge Forum, to create 
mutually reinforcing theory-building discourse online and face-to-face. 

Trait 3: Role of Ideas in Scientific Inquiry 

Examination of Trait 3 revealed a main effect of Class, F(l, 17) = 11.88, p < .01, showing that the 
scores were significantly higher in the Knowledge Building class (M = 1.74, SD = .33) than in the 
comparison class (M = 1.21, SD = .40). A significant effect for Interview, F(l, 17) = 4.48, p < .05, 
indicated that understanding of the role of ideas significantly increased from the pre-interview 
(M = 1.38, SD = .42) to the post-interview (M = 1.54, SD = .47). Again, no Class x Interview 
interaction was observed, F(l, 17) < 1, indicating no significant difference in gains between the 
two classes. 

Examination of the students' responses pertaining to Trait 3 again showed that, in contrast to 
the comparison class. Knowledge Building students recognized the role of ideas in scientific 
progress from the very beginning of the study. For example, one child stated in the pre¬ 
interview: "If no one makes guesses in the world, we would not have all this stuff I" The 
students in the Knowledge Building class further defined discoveries as a result of "right 
questions" and "experimenting through knowledge," suggesting that even young children can 
understand the importance of ideas and experiments in facilitating scientific progresses. One 
child in the post-interview stated: 

Normally scientists have a theory or facts that they base their experiment on, and they 
should probably... have a general idea about what they want to find instead of just kind 
of doing it. 

Another student in the Knowledge Building class said that in order to get unstuck scientists 
should " get a new theory" and "base what they are gonna do next on this new theory." These 
students seem to understand the necessity of reconsidering their theoretical framework if it 
lacks predictive power or things are not advancing, leading to their ability to exceed level 2 
understanding on some occasions. None of the students from the comparison class 
demonstrated this level of understanding. 

How can we explain this advanced understanding of the role of ideas? Work in a Knowledge 
Building classroom tends to focus on basic real-world questions and authentic problems that 
bring students into direct contact with core scientific ideas: What makes it possible for 
airplanes to fly? How does vision work? Why do we have four seasons on earth? Of particular 
importance here is students' formulation of their own research questions and the reformulating 
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of these questions (often radically) as their theorizing proceeds. These questions engage 
students in authentic problems of understanding—things they really wonder about, and these 
then motivate them to find and/or construct provisional accounts that lead to another 
question. Rather then a single-phase, question-answer process they are engaged in ever 
deepening inquires which facilitate theory development and refinement. Knowledge Forum 
supports this culture of continual refinement of "real ideas"—as theories are situated in its 
communal spaces where all students are building on, offering different ideas, questioning 
assumptions. 

Trait 4: Invention 

There was no significant difference between the experimental and comparison group in the 
level of responses to questions about Invention, F(l, 17) < 1, nor did scores improve over time, 
F(1,17) < 1. No significant interaction was observed, F(l, 17) < 1. Thus, the Knowledge Building 
and comparison classes did not significantly differ in the way they described creation of 
innovative objects. Both classes exceeded level 1 conceptions that define "invention" in 
practical terms with one-step procedure (e.g., "scientists put materials together"), and moved 
closer to the level 2 conceptions, describing "invention" as a new thing to the world that is 
produced by means of new ideas, researching and design. 

General Discussion 

The goal of this study was to conduct a "natural experiment" testing the conjecture that a 
theory-building approach is a good way to increase students' understanding of nature of 
science. In order to do this, two intact classes were compared: the experimental class, where 
teaching practices were carried out according to Knowledge Building principles and sustained 
through the Knowledge Forum technology that supports this approach, and a comparison class, 
where an inquiry-based teaching approach prevailed, but without emphasis on creating and 
improving theories. Overall, the Grade 4 students in both classes demonstrated higher scientific 
literacy levels than Grade 7 students from traditional schools, as reported in previous research 
by Carey et al. (1989) and Flonda (1994). Based on the Nature of Science Interview, those 
studies showed that the mean level of 7 th grade students was only 1.0. (to the best of our 
knowledge, there is no data comparable to the Carey et al. and Flonda's data available for 
Grade 4 students). The percentage of students who continuously demonstrated level 2 
conceptions throughout the whole interview was 0% in the Carey et al. study, and 3% in the 
Flonda study. In the present study, by the end of the grade 4, mean scientific literacy level in 
the Knowledge Building class was 1.66 and in the comparison, project-based inquiry class, 1.33. 
The results provide evidence that young students are capable of higher levels of understanding 
of the nature of science than previous research has suggested. 

For evidence bearing on the effects of educational approach we turn to comparison of the 
experimental and the comparison groups, who were similar in demographic characteristics. 
Briefly, the experimental group, taking a theory-building approach, scored significantly higher 
than the comparison group on overall scientific literacy and on three of the four separately 
scored traits. The exception was "invention," which was not a focus of inquiry in either group 
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and which did not show either a significant group difference or significant change from pre- to 
post-test. The groups showed significant gains in overall scientific literacy and on the traits of 
Nature of Theoretical Progress, Theory-Fact Understanding, and Role of Ideas in Scientific 
Inquiry. However, there was no difference between groups in the magnitude of gains, which 
raises question about the causes of the group differences that did emerge. 

Significant differences between the experimental and comparison group were present at pre¬ 
test and differences did not increase over the four-month period of study. In fact, pre-test 
scores of the experimental group were higher than post-test scores of the comparison group on 
all the dependent variables. We will here consider four possible explanations: long-term effects 
of Knowledge Building versus project-based pedagogy, student population differences, school 
environment differences, and differences in technological support. 

Given that most of the participants had long-term exposure to the educational approaches 
pervading their respective schools, it is reasonable to expect that if those approaches differ in 
their effects on scientific literacy, those effects should be evident at the time of pre-test—as 
indeed proved to be the case. That there were not further differences in effect over the course 
of the present study could be attributed to the short time-span of the study (four months) and 
the limited chance for differential effects to stand out against the background of long-term 
educational experience. 

To the extent that there were student population differences, they would tend to favour the 
comparison group. Both schools involved in the study were independent, tuition-charging 
schools, but tuition for the comparison group's school was twice as high, suggesting that the 
students would represent a higher socioeconomic level. The major environmental difference 
was, of course, the fact that the comparison group's school was a girl's school whereas the 
experimental group girls were part of a mixed-gender school and class. According to previous 
research, this should have favored the comparison class. As shown by Guzzetti and Williams 
(1996), grouping girls together generally results in increased participation and feelings of 
confidence for engaging and talking about scientific matters. 

The effects of technology on observed group differences are hard to separate out. Both groups 
made ample use of digital information resources and use of computers for presenting their 
findings. However, the experimental group also used Knowledge Forum, which provided 
"scaffolds" specifically designed to support theory building, as well as a number of other 
facilities for collaborative Knowledge Building (Scardamalia, 2002). We would argue, however, 
that Knowledge Forum cannot be treated as a separate variable potentially affecting results. As 
other papers in this special issue demonstrate. Knowledge Forum is intimately tied to 
Knowledge Building pedagogy. On one hand, from what we observed of activity in the 
experimental group's classroom, virtually all of the sustained effort at theory building took 
place in Knowledge Forum and would not have happened without it. On the other hand. 
Knowledge Forum by itself, without an accompanying Knowledge Building pedagogy and ethos, 
would have only limited applicability to other educational approaches. A common 
recommendation for implementing project-based learning is to use a variety of tools suited to 
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the particular projects undertaken (National Foundation for the Improvement of Education, 
2009). This in essence is what was done in the comparison class, and so there seems to be little 
promise in treating technology as an independent variable in explaining findings of the present 
research. 

Thus we are left with technologically supported Knowledge Building as the most plausible 
explanation of the higher levels of scientific literacy shown by the experimental group. With 
extended immersion in a Knowledge Building environment, nine and ten year old girls were 
able to understand that the goal of science is to improve available explanations of phenomena, 
rather than to accumulate a certain number of facts. They had begun to understand the 
importance of theory in scientific progress. The focus on ideas—as compared to focus on facts 
and activities—allows teachers and students to rise above the existing standards of learning 
and construct deep understanding of phenomena. Science becomes for students an exciting 
enterprise, rather than a plodding one-thing-after another exercise. 

The present research offers empirical backing for a way to help elementary school students 
come to understand science as a theory-driven process. This should complement and improve 
upon scientific literacy agendas making their way into curriculum standards and teacher 
resource material. Governments around the world are looking for ways to build capacity for 
knowledge creation and innovation. They look to education to play a vital part in advancing 
scientific literacy, but the attempt to teach a host of related skills is drawing criticism, as 
solutions involve adding more skills to an already crowded curriculum. The Knowledge Building 
pedagogies and technologies reported provide alternatives to that standard solution, along 
with new tools to inform practice as it proceeds. The research also suggests classroom work 
that facilitates creative work in knowledge advancement for which all students share 
responsibility. Overall the research and material to inform teacher development should add to 
the currently meager knowledge base for designing ways by which education might contribute 
to increasing societal capacity for scientific literacy. 
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